Decision Trees: More Theoretical Justification

We study impurity-based decision tree algorithms such as CART, C4.5, etc., so as to better understand their theoretical underpinnings. We consider such algorithms on special forms of functions and distributions. We deal with the uniform distribution and functions that can be described as unate functions, linear threshold functions and readonce DNF. For unate functions we show that that maximal purity gain and maximal influence are logically equivalent. This leads us to the exact identification of unate functions by impurity-based algorithms given sufficiently many noise-free examples. We show that for such class of functions these algorithms build minimal height decision trees. Then we show that if the unate function is a read-once DNF or a linear threshold functions then the decision tree resulting from these algorithms has the minimal number of nodes amongst all decision trees representing the function. Based on the statistical query learning model, we introduce the noisetolerant version of practical decision tree algorithms. We show that when the input examples have small classification noise and are uniformly distributed, then all our results for practical noise-free impurity-based algorithms also hold for their noise-tolerant version.